library(rollama)library(tibble)library(purrr)q <-tribble(~role, ~content,"system", "You assign texts into categories. Answer with just the correct category.","user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative")query(q)#> #> ── Answer ────────────────────────────────────────────────────────#> Category: Negative
Also packages for Dart, Swift, C#, Java, PHP, Rust etc.
Python:
import ollamaresponse = ollama.chat(model='llama2', messages=[ {'role': 'system','content': 'You assign texts into categories. Answer with just the correct category.', }, {'role': 'user','content': 'text: the pizza tastes terrible\ncategories: positive, neutral, negative', },])print(response['message']['content'])#> Category: Negative
JavaScript:
import ollama from'ollama'const response =await ollama.chat({model:'llama2',messages: [ {'role':'system','content':'You assign texts into categories. Answer with just the correct category.', }, {'role':'user','content':'text: the pizza tastes terrible\ncategories: positive, neutral, negative', }, ],})console.log(response.message.content)#> Category: Negative
Classification–One-shot
q <-tribble(~role, ~content,"system", "You assign texts into categories. Answer with just the correct category.","user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative","assistant", "Category: Negative","user", "text: the service is great\ncategories: positive, neutral, negative")query(q)#> #> ── Answer ────────────────────────────────────────────────────────#> Category: Positive
Neat effect: change the output structure
q <-tribble(~role, ~content,"system", "You assign texts into categories. Answer with just the correct category.","user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative","assistant", "{'Category':'Negative','Confidence':'100%','Important':'terrible'}","user", "text: the service is great\ncategories: positive, neutral, negative")answer <-query(q)#> #> ── Answer ────────────────────────────────────────────────────────#> {'Category':'Positive','Confidence':'100%','Important':'great'}
Classification–Few-shot
q <-tribble(~role, ~content,"system", "You assign texts into categories. Answer with just the correct category.","user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative","assistant", "Category: Negative","user", "text: the service is great\ncategories: positive, neutral, negative","assistant", "Category: Positive","user", "text: I once came here with my wife\ncategories: positive, neutral, negative","assistant", "Category: Neutral","user", "text: I once ate pizza\ncategories: positive, neutral, negative")query(q)#> #> ── Answer ────────────────────────────────────────────────────────#> Category: Neutral
Classification–Chain-of-Thought
q_thought <-tribble(~role, ~content,"system", "You assign texts into categories. ","user", "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts.")output_thought <-query(q_thought)#> #> ── Answer ────────────────────────────────────────────────────────#> #> Based on the given text, I would assign it to the category of "negative#> sentiment." The use of the word "terrible" and the tone of the sentence convey#> a strong negative emotion towards the taste of the pizza. It's unlikely that#> someone would describe something as "terrible" if they were neutral or#> positive about it.
Now we can use these thoughts in classification
q <-tribble(~role, ~content,"system", "You assign texts into categories. ","user", "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts.","assistant", pluck(output_thought, "message", "content"),"user", "Now answer with just the correct category (positive, neutral, or negative)")query(q)#> #> ── Answer ────────────────────────────────────────────────────────#> #> Negative
Arsenal and Liverpool only two of Prem’s Big Six NOT to reject new Super League plot as football descends i…
1
https://www.rte.ie/sp…
2023-12-21 13:29:36
What does the ECJ Super League ruling mean?
1
https://news.sky.com/…
2023-12-21 08:34:00
European Super League: EU's top court rules FIFA and UEFA acted unlawfully in blocking breakaway competition
1
https://news.sky.com/…
2023-12-21 11:48:00
How would the new European Super League work - and what does it mean for Premier League clubs?
1
https://www.breakingn…
2023-12-21 10:06:12
Judges rule Uefa blocking European Super League was contrary to EU law
1
https://www.theguardi…
2023-12-21 09:01:49
European Super League project gets huge boost after court of justice ruling
1
https://www.thejourna…
2023-12-21 11:21:07
Uefa and Fifa bid to block Super League broke EU law, court rules
1
https://www.theguardi…
2023-12-20 19:25:11
Uefa v European Super League: what could verdict mean for football?
1
https://www.thesun.ie…
2023-12-21 09:09:45
Terrified fans fear European Super League is about to return after European Court of Justice ruling…
1
https://www.thesun.ie…
2023-12-21 11:17:45
European Super League LIVE: Uefa and Fifa lose court case as hated tournament is back on the table – l…
Annotation
library(rollama)eval_set_df_annotated <- eval_set_df |>group_by(cluster) |>summarise(headlines =paste0(title, collapse ="\n\n"),n =length(title)) |>filter(n >1) |>mutate(label =NA)for (i inseq_len(nrow(eval_set_df_annotated))) { q <- tibble::tribble(~role, ~content,"system", "You assign short highly descriptive topic labels to a collection of headlines about the same topic. Answer with just the topic label.","user", "headlines: \nGuns, ammunition, drugs and cash seized as part of investigation into organised crime in Tipperary as...\n\nFirearms, ammunition, machete, crack cocaine, heroin and cash seized in garda operation - Irish Mirror Online\n\nMan arrested after gardaí seize weapons and drugs in Clonmel","assistant", "blow against organised crime","user", "headlines: Taoiseach Leo Varadkar stung by Russian pranksters in hoax diplomatic call\n\nTaoiseach Leo Varadkar ‘pranked’ in phone call by pro-Kremlin Russian ‘comedians’ pretending to be African diplomats\n\nGovernment changes meetings protocol after Taoiseach duped in Russian hoax call\n\nTaoiseach Leo Varadkar ‘pranked’ in phone call by pro-Kremlin Russian ‘comedians’ pretending to be African diplomats\n\nLeo Varadkar falls victim to a prank by two Russian comedians\n\nLeo Varadkar insists he 'played along' with Russian pranksters on hoax video call - Irish Mirror Online\n\nLeo Varadkar falls victim to prank call by pair of Russian comedians - Irish Mirror Online\n\nTaoiseach was 'suspicious immediately' of prank by two Russian comedians but had to play along\n\nRed-faced Leo Varadkar makes urgent change to Government video call security protocols after embarrassing...\n\nI played along with prank call by Russian comedians – Taoiseach\n\nTaoiseach Leo Varadkar ‘played along’ with Russian comedians’ prank phone call after he became suspicious\n\nTaoiseach Leo Varadkar ‘played along’ with Russian comedians’ prank phone call after he became suspicious\n\nLeo Varadkar “pranked” by pro-Kremlin Russian “comedians”","assistant", "Taoiseach pranked by Russians","user", "headlines: Social media platform X back online after global outage\n\nSocial media platform X back online after global outage\n\nX is down: Thousands of users experiencing outages on platform formerly known as Twitter\n\nTwitter down: X website owned by Elon Musk suffers outages as thousands of users report problems...\n\nElon Musk’s X back online after global outage\n\nSocial media platform X back online after global outage · TheJournal.ie\n\nX users experiencing widespread outages","assistant", "platform X outage","user", glue::glue("headlines: {eval_set_df_annotated$headlines[i]}") ) eval_set_df_annotated$label[i] <-query(q, screen =FALSE) |>pluck("message", "content")}
version:'3.6'services: # ollama and APIollama:image: ollama/ollama:latestcontainer_name: ollamapull_policy: missingtty:truerestart: unless-stopped # Expose Ollama API outside the container stackports:- 11434:11434volumes:- ollama:/root/.ollama # GPU support (turn off by commenting with # if you don't have an nvidia gpu)deploy:resources:reservations:devices:-driver: nvidiacount:1capabilities:- gpu # webui, nagivate to http://localhost:3000/ to useollama-webui:image: ghcr.io/ollama-webui/ollama-webui:maincontainer_name: ollama-webuipull_policy: missingvolumes:- ollama-webui:/app/backend/datadepends_on:- ollamaports:- 3000:8080environment:-"OLLAMA_API_BASE_URL=http://ollama:11434/api"extra_hosts:- host.docker.internal:host-gatewayrestart: unless-stoppedvolumes:ollama:{}ollama-webui:{}
tinyurl.com/ollama-gist
What I will use here
tinyurl.com/r-ollama
Issues
Methodological:
field is still an academic Wild West – lack of established standards for the reliable, reproducible and ethical use of gLLMs (see Törnberg draft)
open models still lack behind proprietary ones (but not as far as we though a year ago!)
consistent bias (as opposed to sporadic bias with a group of human annotators)
Technical:
Small-ish hurdle to configure GPU, especially with Docker (and cost to get (NVIDIA*) GPU)
In short: it’s still work in progress (but impressive nonetheless!)
References
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.”Proceedings of the National Academy of Sciences 120 (30). https://doi.org/10.1073/pnas.2305016120.
Heseltine, Michael, and Bernhard Clemm von Hohenberg. 2023. “Large Language Models as a Substitute for Human Experts in Annotating Political Text.” OSF. https://doi.org/10.31219/osf.io/cx752.
Spirling, Arthur. 2023. “Why Open-Source Generative AI Models Are an Ethical Way Forward for Science.”Nature 616 (7957): 413–13. https://doi.org/10.1038/d41586-023-01295-4.
Törnberg, Petter. 2023. “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.” arXiv. http://arxiv.org/abs/2304.06588.
Weber, Maximilian, and Merle Reichardt. 2023. “Evaluation Is All You Need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer Using Open Models.”https://arxiv.org/abs/2401.00284.
Zhong, Qihuang, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. “Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-Tuned BERT.” arXiv. http://arxiv.org/abs/2302.10198.